Social world knowledge: Modeling and applications

Social world knowledge is a key ingredient in effective communication and information processing by humans and machines alike. As of today, there exist many knowledge bases that represent factual world knowledge. Yet, there is no resource that is designed to capture social aspects of world knowledge. We believe that this work makes an important step towards the formulation and construction of such a resource. We introduce SocialVec, a general framework for eliciting low-dimensional entity embeddings from the social contexts in which they occur in social networks. In this framework, entities correspond to highly popular accounts which invoke general interest. We assume that entities that individual users tend to co-follow are socially related, and use this definition of social context to learn the entity embeddings. Similar to word embeddings which facilitate tasks that involve text semantics, we expect the learned social entity embeddings to benefit multiple tasks of social flavor. In this work, we elicited the social embeddings of roughly 200K entities from a sample of 1.3M Twitter users and the accounts that they follow. We employ and gauge the resulting embeddings on two tasks of social importance. First, we assess the political bias of news sources in terms of entity similarity in the social embedding space. Second, we predict the personal traits of individual Twitter users based on the social embeddings of entities that they follow. In both cases, we show advantageous or competitive performance using our approach compared with task-specific baselines. We further show that existing entity embedding schemes, which are fact-based, fail to capture social aspects of knowledge. We make the learned social entity embeddings available to the research community to support further exploration of social world knowledge and its applications.

"… We make the SocialVec framework and the resulting entity embeddings as learned and applied in this work accessible to the research community, and believe that this has the potential of making a significant contribution to exploring social world knowledge as reflected in Twitter. Our code is available at github.com/nirlotan/SocialVecTraining. Another respository that contains the entity embeddings, as well as an API for assessing entity similarity, is available at https://github.com/nirlotan/SocialVec." Q: Author could mention the selection bias regarding the model predicting personality features of the users. It could be that demographics are correlated with related information users provide on Twitter e.g., younger users provide more info on their bio thus MTurk work could do a better job in judging their demographics.
R: We agree that there could be a selection bias in labeling the socio-demographic characteristics of users by crowd workers. Unfortunately, there is no alternative labeling method that is bias-free. For example, asking users to actively provide their details would only attract a subset of the population; or, labeling users based on their selfdescriptions on Twitter would be inherently biased to those users who tend to share personal information online. That being said, we kindly note that the dataset used in our experiments, which was labeled by MTurk crowd workers, was collected and annotated prior to our work by other researchers (Volkova et al.). It is a public dataset, for which there exist previously published results using alternative methods, which we compare our results against. Nevertheless, to clarify the point raised, we added the following text to Section `6.1 Dataset': "We note that beyond the labels being subjective, the dataset may present a selection bias, e.g., due to varying levels of self-exposure on social media by different subpopulations. Yet, we evaluate multiple alternative methods using the same data, and in the same conditions, where this forms a viable evaluation setup." In addition, we discuss potential general biases involved in automatically predicting the personal traits of users in Sections `7.2 Limitations' and `7.3 Ethical considerations'.
Q: The paper is currently slightly too long particularly the abstract is wordy and hard to follow.
R: Thank you for this comment. We have made a substantial effort to make the whole paper less wordy and more fluent. Overall, this effort resulted in a reduction of the paper length by more than 10%. In particular, we have edited the abstract, having it shortened as a result by roughly 25% (from 368 to 283 words). A document that highlights the tracked changes is submitted as part of this revision, illustrating the many edits performed to the revised manuscript.

REVIEWER 2:
The paper proposes an interesting study and it proposes SocialVec. The topic of the paper is really interesting, however it is not clear how the evaluation part has been organized. More details should be included, in particular the section related to the comparison. It is completely unclear the choices concerning the comparison. Please clarify this point in order to understand if this section is important or not, as it is.
R: Thank you for noting this issue. To clarify the setup of the evaluation, and the underlying motivations for this setup, we have created a new main section in the revised paper (Sec.4) which is titled `4. Evaluation'.
We then added Section `4.1. Evaluation methodology', which includes two main paragraphs. The first paragraph discusses `intrinsic vs. extrinsic evaluation'. Below are some excerpts from this paragraph: "Unfortunately, there are no relevant human-judged benchmarks that assess entity similarity. Instead, we take a direct look at inter-entity similarities by exploring the entities that are most similar to example entities of interest in the learned social embedding space. …. This intrinsic evaluation is presented and discussed in Section 4.2. Nevertheless, we mainly place our focus on extrinsic evaluation, where we gauge the utility of the learned social entity embeddings for end applications which involve social inference. First, we assess the political bias of news sources in terms of entity similarity in the social embedding space. Second, we predict the personal traits of individual Twitter users based on the social embeddings of entities that they follow. These studies are presented in Sections 5 and 6, respectively." The second paragraph in Section 4.1 discusses our comparison against alternative methods and baselines. Here is an excerpt from this paragraph: "In evaluating SocialVec on end tasks, we review and compare our results against alternative methods, which have been designed and applied per those target tasks and experimental datasets. In addition, we contrast our social entity embeddings with existing entity embedding schemes, namely, Wikipedia2Vec and Wikidata RBG embeddings. … To the extent that SocialVec achieves higher performance on tasks of social inference, this implies that it is superior in capturing dimensions of social meaning." We believe that this section improves the understanding of how we apply and evaluate the proposed framework. .